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Abstract 

The purpose of this study was to determine if an experimental 5-item measure of barriers to 
implementing and sustaining school-wide prevention practices, the Assessment of Barriers to 
Implementation and Sustainability in Schools (ABISS), would relate to objective measures of 
school-wide positive behavioral interventions and supports (SWPBIS) implementation fidelity. 
The ABISS was administered to individuals in 704 U.S. schools implementing SWPBIS across 
11 states, and scores were compared to school demographic variables and SWPBIS fidelity of 
implementation. Results showed acceptable model fit for the ABISS, partial measurement 
invariance, and statistically significant relations, after controlling for school demographic 
variables, with SWPBIS fidelity of implementation, except for schools implementing SWPBIS 
for 5 or more years. These patterns indicate substantial but decreasing relations of perceived 
barriers to implementation fidelity as schools continue to implement SWPBIS. Implications are 
discussed in terms of how assessment of perceived barriers can support implementation and 
sustainability of school-wide prevention practices. 

Keywords: prevention, implementation science, barriers, fidelity of 


implementation, positive behavior support 



BARRIERS TO SUSTAINABILITY OF PREVENTION PRACTICES 


3 


Examining Barriers to Sustained Implementation of School-wide Prevention Practices 

Because schools, districts, and states often adopt and implement effective educational 
practices that are later abandoned, identifying variables that support the sustainability of 
educational practices is a growing area of research (McIntosh, Horner, & Sugai, 2009). In 
research on sustainability of educational practices, the following terms should be differentiated: 
implementation fidelity, sustained implementation, and sustainability. We consider 
implementation fidelity to be the extent to which trained interventionists deliver the intervention 
consistently, and as intended (Sanetti & Kratochwill, 2009). As an educational practice continues 
to be implemented over a period of several years with adequate fidelity and positive student 
outcomes, sustained implementation is evident. In contrast, sustainability is the potential of 
sustained implementation, based on aspects of the practice, its implementation, its context, and 
the systems in place to support it (McIntosh & Turri, 2014). To meet the goal of sustained 
implementation, schools need to engage in an iterative process of responding to changes in 
context (McIntosh, Horner, et al., 2009). Barriers to the sustainability of school-based practices 
are inevitable, often arising from initial levels or changes to the structure and dynamics of 
implementing schools in areas such as resources, staff capacity, and district policy. When an 
intervention can be implemented with integrity despite these potential barriers, it can be called 
sustainable (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). 

In terms of potential for sustained implementation, the current level of fidelity is 
particularly important. When fidelity is high for an evidence-based practice, outcomes valued by 
school staff may be more likely to be achieved, and momentum and motivation of school staff to 
maintain a high level of implementation may be increased (McIntosh, Horner, et al., 2009). 

When the impact of the intervention is visible, i.e., positive changes are observed, staff members 
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may associate their efforts with the outcomes, which can help facilitate their continued effort 
(Andreou, McIntosh, Ross, & Kahn, 2015; Forman, Olin, Hoagwood, Crowe, & Saka, 2009; Han 
& Weiss, 2005). Correspondingly, sustained implementation may be threatened if staff perceive 
that the intervention is not leading to important outcomes or worth their effort (Lohrmann, 
Martin, & Patil, 2013). For these reasons, determining the extent to which perceived barriers to 
sustained implementation relates to implementation fidelity is an important first step toward 
identifying the effect of perceived barriers on sustained implementation. However, there is 
currently no known quantitative measure of barriers available. 

In the current study, we present a preliminary, brief measure of school personnel 
perceived barriers to sustained implementation, and examine the extent to which the measure 
relates to concurrent measures of School-wide Positive Behavioral Interventions and Supports 
(SWPBIS) implementation fidelity. SWPBIS is a systems-level approach to implementing 
effective behavior support practices focusing on instruction and environmental change to 
enhance prosocial behaviors and reduce problem behaviors across all school settings and for all 
students (Sugai et al., 2010). Identifying and addressing potential barriers to sustained 
implementation is particularly important for school-wide interventions such as SWPBIS, given 
its widespread dissemination across diverse school contexts and its well documented positive 
effects on multiple student outcomes (Homer et al., 2009; McIntosh, Bennett, & Price, 2011). 

In its current form the scale is not intended for use as a diagnostic tool, rather the measure 
is intended to be an efficient measure of overall levels of barriers to sustainability. SWPBIS was 
selected because of its wide implementation and sustainability (McIntosh, Filter, Bennett, Ryan, 
& Sugai, 2010), making it an opportune focus of systems change research. Although the present 
study focuses on SWPBIS, barriers to implementation are inevitable in any large-scale 
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implementation effort; therefore, the implications of this study could potentially inform other 
school-based practices, and implementation science more broadly. 

Failure to Sustain Educational Practices: What We Know About Barriers 

Several broad factors have been discussed as potential barriers to sustained 
implementation of school-based practices, including academic, behavioral, and mental health 
initiatives. These identified barriers informed the development of the brief measure of perceived 
barriers to implementation. The majority of studies in this area are case studies of failures to 
sustain and analyses of school personnel and stakeholder’s perceptions of barriers through 
qualitative interviews and surveys (e.g., Bambara, Goh, Kern, & Caskie, 2012; Forman et al., 
2009; Lohrmann, Forman, Martin, & Palmieri, 2008; Sindelar, Shearer, Yendol-Hoppey, & 
Liebert, 2006). As such, the majority of the knowledge base regarding contextual barriers to 
sustained implementation of educational practices is based on descriptive, retrospective accounts. 
The lack of quantitative analysis of contextual barriers to sustainability may erroneously 
implicate educator motivation as the primary reason for failures to sustain. Specific barriers that 
have been identified, predominately based on studies of school personnel and stakeholder 
perceptions, are described below. Because these studies have poorly differentiated between the 
concepts of implementation and sustainability, we discuss the concepts together, given their 
logical relation. 

Administrator Support 

As the head of the school, administrators are instrumental in setting the priority for new 
interventions (Handler et al., 2007). As such, lack of administrator support is frequently reported 
as a perceived barrier to successful implementation of school-wide and individual positive 
behavior support practices (Andreou et al., 2015; Bambara, Nonnemacher, & Kern, 2009; 



BARRIERS TO SUSTAINABILITY OF PREVENTION PRACTICES 


6 


Kincaid, Childs, Blase, & Wallace, 2007; Lohrmann et al., 2008). Lohrmann et al. (2013) 
reported that implementers of SWPBIS perceived that lack of active administrator support might 
negatively impact staff buy-in by sending a mixed message to staff regarding their need to 
participate. Developers of evidence-based mental health and prevention practices in schools 
have indicated that in their experience, even passive administrator resistance, such as generally 
acknowledging support but failing to participate, may be detrimental to the success of a practice 
(Forman et al., 2009). 

Staff Buy-in 

Obtaining staff buy-in for a practice is an essential component of successful 
implementation. Aside from the logistical challenge of recruiting staff with relevant skills, lack 
of staff buy-in due to negative beliefs regarding the practice is a perceived barrier commonly 
reported in research. For example, staff may resist changing their personal practices based on 
either a desire to maintain the status quo or lack of interest in the intervention itself (Forman et 
al., 2009). Qualitative interview and survey studies consistently indicate the perceived difficulty 
of obtaining staff buy-in when the philosophical principles of a practice are in opposition to 
existing staff beliefs (Andreou et al., 2015; Bambara et al., 2012; Bambara et al., 2009; Kincaid 
et al., 2007; Lohrmann et al., 2008; Pinkelman, McIntosh, Rasplica, Berg, & Strickland-Cohen, 
2015). For example, in SWPBIS, staff may resist the principle of acknowledging students for 
appropriate behavior if they believe punishment to be more effective in reducing problem 
behavior (Lohrmann et al., 2013). 

Staff Capacity to Implement 

Because school staff are ultimately the individuals who deliver a practice, barriers to staff 
capacity to implement are important to address. Previous interview and survey research on 
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school-based behavior practices indicate that staff capacity is diminished by inadequate training, 
lack of ongoing technical assistance, and too few staff members trained (Bambara et al., 2012; 
Bambara et al., 2009; Kincaid et al., 2007). Inadequate training may threaten program success 
from the outset; Kratochwill, Volpiansky, Clements, and Ball (2007) suggest that strong training 
and ongoing professional development are essential components for ensuring that school 
personnel implement with fidelity. Qualitative interview and survey studies indicate that the 
provision of adequate time to complete intervention activities during the school day is also 
perceived as important (Forman et al., 2009; Pinkelman et al., 2015), as is the perception that the 
intervention can efficiently align with regular routines and not be perceived as an additional 
burden (Andreou et al., 2015; Bambara et al., 2009; Coffey & Homer, 2012). 

Turnover 

Staff turnover can affect both administrator support and staff capacity to implement. 
Given that administrator support has been noted as essential to successful implementation, 
administrator turnover may be particularly challenging. For example, in a case study analysis of 
failure to sustain a school inclusion practice that had been successfully implemented, Sindelar et 
al. (2006) found that administrator turnover was reported as a key contributor to the failure, 
noting that the new administrator had different priorities and showed less support for the 
practice. In terms of staff turnover, departing champions of the practice (i.e., key staff members 
invested in the practice), and incoming untrained staff may all contribute to diminished 
implementation. Sindelar and colleagues suggested that both types of staff turnover diluted 
program acceptance and capacity to implement. One specific challenge with staff turnover is the 
need to train new staff. When staff turnover is high, schools may use substantial resources to 
establish buy-in and provide training for new staff, contributing to a sense of being “stuck” in a 
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perpetual state of initial implementation (Hatch, 2000). 

Competing Initiatives 

Implementing a new practice in a school is inherently difficult given that schools are 
faced with concurrent implementation of multiple, possibly competing, initiatives to satisfy 
needs and policies at multiple levels (e.g., school initiatives vs. state initiatives). Sanford 
DeRousie and Bierman (2012) found that teachers most frequently reported competing initiatives 
as a barrier to the sustainability of an evidence-based literacy and social emotional preschool 
curriculum. As a result, they warned against implementing a new initiative without also 
addressing strategies to reduce competing priorities. Langley, Nadeem, Kataoka, Stein, and 
Jaycox (2010) also reported that competing initiatives was the most frequently endorsed barrier 
among clinicians who were unsuccessful in implementing a school-based mental health 
initiative. In schools, one major barrier is the focus on standardized academic testing stemming 
from federal and state initiatives, which may be perceived as superseding non-academic goals, 
threatening non-academic interventions in particular (Forman et al., 2009). 

Barriers to Sustainability of School-wide Positive Behavioral Interventions and Supports 

Two recent studies have examined perceptions of school staff regarding facilitators and 
barriers to sustained implementation of SWPBIS. McIntosh et al. (2014) analyzed respondents’ 
perceived importance of contextual features for implementation and sustainability of SWPBIS 
using a mixed-methods approach. Qualitative analyses revealed that limited resources (including 
time and funding) were identified as the most significant barrier to sustaining SWPBIS; other 
frequently endorsed barriers included high rates of staff turnover, low fidelity of implementation, 
and insufficient staff buy-in. Pinkelman et al. (2015) aimed to replicate and expand the results of 
McIntosh et al., using a larger and more diverse sample. Also based on qualitative analyses, the 
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most frequently reported barriers were limited staff buy-in, followed by insufficient resources, 
both temporal (i.e., time to do SWPBIS activities) and monetary (i.e., funding). 

Although the results of these studies shed light on the perspectives of school team 
members regarding barriers to sustained implementation, both studies, together with the majority 
of the studies previously reviewed, relied largely on descriptive interviews or surveys to identify 
barriers to sustainability. Across these studies, there is tremendous consistency among the 
perceptions of school personnel; however, there have been limited efforts to identify barriers to 
implementation using objective measures of fidelity or sustained implementation. For example, 
McIntosh et al. (2014) reported that raters’ perceived lack of implementation fidelity to be a 
significant barrier to sustainability, which warrants empirical corroboration with fidelity of 
implementation data. 

Additionally, it is important to consider the schools’ stage of SWPBIS implementation. 
Similar to other school-based interventions, SWPBIS is a systems-level approach, requiring 
several years to achieve full implementation (Sugai, Homer, & McIntosh, 2008). Across each 
stage of implementation, there may be different barriers to continued implementation. In the 
current study, three stages were conceptualized, representing schools implementing for 0 to 1 
years (initial implementation), 2 to 4 years (full operation), and 5 or more years (sustainability; 
Adelman & Taylor, 1997; Mercer, McIntosh, Strickland-Cohen, & Homer, 2014). Both 
McIntosh et al. (2014) and Pinkelman et al. (2015) included schools across all stages of 
implementation in their samples; however, it is important to systematically examine differences 
in reported barriers, and the extent to which relations between reported barriers and fidelity differ 
by stage of implementation. 


Purpose of the Study 
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Studying perceived barriers to sustained implementation across all stages of SWPBIS 
implementation will provide critical information to the emerging literature on the sustainability 
of school-based practices, which has to date been largely anecdotal. The purpose of this study is 
to evaluate the factor structure of a brief measure of SWPBIS school team members’ perceived 
barriers to sustained implementation, and the extent to which the measure relates to fidelity of 
implementation data at each stage of implementation. 

The following research questions were examined: 

1. What is the factor structure of the barriers measure, and is it invariant across stages of 
implementation? 

We expected the barriers scale, as a brief measure, to be unidimensional. Testing measurement 
invariance (i.e., similarity of scale psychometrics across stages) permitted investigation of 
substantive research questions regarding differences in barriers across stages of implementation. 
Without establishing measurement invariance, observed mean differences in perceptions of 
barriers could be an artifact of psychometric differences in the barriers measure across SWPBIS 
stages of implementation. 

2. Are there mean differences in perceptions of barriers across schools at different SWPBIS 
stages of implementation? 

Because sustained implementation, in part, requires that schools overcome barriers to 
implementation, we expected there to be fewer perceived barriers in latter stages of 
implementation. 

3. Controlling for school demographic variables, to what extent do perceptions of barriers 
relate to fidelity of SWPBIS implementation at each implementation stage? 
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We considered it important to account for school demographic control variables, such as school 
level (i.e., elementary vs. middle or high schools) and the percentage of students in the school 
eligible for free or reduced price meals, because prior research has indicated that these variables 
relate to sustainability (McIntosh, Kim, Mercer, Strickland-Cohen, & Horner, 2015) or sustained 
implementation (McIntosh, Mercer, Nese, Strickland-Cohen, & Hoselton, 2015). 

Method 

Participants and Settings 

A total of 704 U.S. K-12 public schools (within 11 states and spanning all four regions of 
the US Census Bureau) participated in the study. These schools were part of a larger sample of 
schools participating in a longitudinal study examining implementation and sustainability of 
SWPBIS. This subsample included all schools with complete demographic data available from 
the National Center for Educational Statistics (NCES). Demographic data for the schools are 
summarized in Table 1. All schools were currently implementing or preparing to implement 
SWPBIS in the next school year. Of these schools, 180 (25%) were in the initial implementation 
stage (0 to 1 years), 329 (47%) were in the full operation stage (2 to 5 years), and 195 (28%) 
were in the sustainability stage (5 or more years). 

Measures 

Barriers to Sustainability. The Assessment of Barriers to Implementation and 
Sustainability in Schools (ABISS) is an experimental 5-item scale with 4 response options (1 = 
Not True ; 4 = Very True) assessing potential barriers to implementation or sustainability of 
school-wide interventions. The items in the ABISS (see Table 2) were initially developed as part 
of the School-wide Universal Behavior Sustainability Index- School Teams (SUBSIST; 

McIntosh, Doolittle, Vincent, Homer, & Ervin, 2009). The SUBSIST is a research-validated 



BARRIERS TO SUSTAINABILITY OF PREVENTION PRACTICES 


12 


measure used to assess the critical features that facilitate or hinder the sustainability of school- 
based universal behavior support practices. Internal consistency of the ABISS in the current 
sample is reported in the results section; based on data reported in McIntosh, MacKay, et al. 

(2011), inter-rater reliability (weighted kappa = .83) and 3-week test-retest reliability (weighted 
kappa = .86) for ABISS items was high. Evidence for content validity, based on data reported in 
McIntosh, MacKay, et al. (2011), included an expert panel review of the measure, the response 
format, and the extent to which the specific items assessed a potential barrier to sustainability. 
Results showed a content validity index score of .95, indicating strong content validity (Rubio, 
Berg-Wagner, Tebb, Lee, & Rauch, 2003). Percentages of missing data on individual ABISS 
items ranged from 2.2 to 6.7%. 

SWPBIS Fidelity of Implementation. Fidelity of implementation of SWPBIS at each 
school was assessed using the following research-validated measures: the School-wide 
Evaluation Tool (SET; Sugai, Lewis-Palmer, Todd, & Homer, 2001), the School-wide 
Benchmarks of Quality (BOQ; Kincaid, Childs, & George, 2005), the PBIS Self Assessment 
Survey (SAS; Sugai, Horner, & Todd, 2000), and the Team Implementation Checklist (TIC; 
Sugai, Todd, & Horner, 2001). 

School-wide Evaluation Tool. The SET is completed during a site visit from a trained 
external evaluator. Psychometric analyses revealed strong internal consistency (a = .96), test- 
retest reliability (mean = 97%) and inter-rater agreement (mean agreement = .99; Horner et al., 
2004). SET implementation average scores were available for 409 schools (58% of sample). 

School-wide Benchmarks of Quality. The BOQ is also an external evaluation that is 
based on observation and permanent product review. Analyses of the measure’s psychometric 
properties indicated strong internal consistency (a = .96), test-retest reliability (mean = 97%), 



BARRIERS TO SUSTAINABILITY OF PREVENTION PRACTICES 


13 


inter-rater reliability (mean agreement = 89%), and concurrent validity (correlation with the SET, 
r = .51; Cohen, Kincaid, & Childs, 2007). BOQ total ratio scores were available for 348 schools 
(50% of sample). 

PBIS Self Assessment Survey. The SAS is a self-assessment fidelity of implementation 
measure. Its correlation with the SET has been documented as .75 (Horner et al., 2004). 
Psychometric analysis showed strong internal consistency for the school-wide subscale and total 
scores of the measure (a range = .75 to .94; Hagan-Burke et al., 2005; Safran, 2006). SAS 
implementation average scores were available for 461 schools (65% of sample). 

Team Implementation Checklist. The TIC is another self-report fidelity measure. The 
TIC has demonstrated strong internal consistency (ordinal a = .94; McIntosh, Mercer, et al., 
2015). TIC implementation average scores were available for 284 schools (40% of sample). 
Procedure 

Schools implementing or preparing to implement SWPBIS were recruited through the 
schools’ state SWPBIS networks through email and sharing contact information at state or 
district conferences. The invitation requested that any one member of each school’s SWPBIS 
team could complete the ABISS online for their respective school, between fall 2012 and spring 
2013. In the case of multiple responses per school, the survey with the least missing data was 
selected as the response for the school. Of the respondents, 63% were school PBIS team leaders, 
facilitators, or internal coaches; 23% were school administrators; 8% were other school faculty 
or staff members; and 5% were external to the school (e.g., a district PBIS team facilitator). 
ABISS data were then electronically linked to the schools’ PBIS fidelity data (i.e., the most 
recent SET, BOQ, SAS, and/or TIC scores for the school year the ABISS was completed), 
reported to the National TA Center on PBIS through a secure website (www.pbisapps.org) . 
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Data Analyses 

Prior to testing mean differences on the ABISS by stages of implementation, the factor 
structure and measurement invariance of the ABISS was first investigated (Research Question 
#1). Because items on the ABISS have a four-option response format and responses to items 
were positively skewed (range: 0.82 to 2.47), ABISS item responses were specified as ordered 
categorical, and the mean- and variance-adjusted weighted least squares (WLSMV) estimator 
with the theta parameterization was used. First, the unidimensionality of the ABISS within each 
implementation group was investigated by comparing the magnitude of the first relative to the 
second eigenvalues in ordinal exploratory factor analyses (EFA). After establishing 
unidimensionality within each implementation group, the internal consistency of the ABISS was 
assessed using ordinal alpha (Zumbo, Gadermann, & Zeisser, 2007). To test measurement 
invariance across implementation groups, a configural invariance model with factor loadings and 
item thresholds free to vary across groups was tested against a scalar invariance model with 
factor loadings and item thresholds constrained to equality across groups. In the configural 
invariance model, item residuals were constrained to equal one and ABISS factor means were 
constrained to equal zero for model identification. In the scalar invariance model, for the initial 
implementation group the factor mean was constrained to equal zero and item residual variances 
were constrained to equal one for model identification, but factor means and item residual 
variances were freely estimated in the institutionalization and sustainability groups. Change in 
model fit was tested using the DIFFTEST option in M plus (Muthen & Muthen, 2012), with a 
statistically significant chi-square value indicating that scalar invariance did not hold. Following 
a statistically significant chi-square difference test, potential sources of non-invariance were 
investigated by examining model modification indices and freeing loadings and thresholds one 
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item at a time until a partially invariant model that did not statistically differ from the configural 
invariance model was found. Following the tests of measurement invariance of the ABISS, mean 
differences by implementation group on the ABISS were examined in the final partial invariance 
model (Research Question #2). 

To enable testing of the relation between ABISS scores and fidelity (Research Question 
#3), we first tested the appropriateness of fitting a latent fidelity construct with SET, BOQ, SAS, 
and TIC scores as indicators. Scores on each of the fidelity measures were proportions with 
negative skew (range: -.80 to -2.42). To better meet model assumptions, angular transformations 
(i.e., arcsine root; see Maindonald & Braun, 2010, p. 279) were used on the scores from each 
fidelity measure, resulting in much less negative skew (range: -.12 to -.72). Following 
transformation, the reliability of the fidelity factor was examined by calculating a composite 
reliability estimate (Raykov, 1997). Then, the measurement invariance of the factor was 
examined across implementation groups. Unlike the ordered categorical ABISS responses, scores 
on the fidelity measures were continuous; consequently, the robust maximum likelihood 
estimator (MLR) was used, and configural (free factor loadings and intercepts), metric (equal 
loadings), and scalar (equal loadings and intercepts) models were tested. Differences in model fit 
across the invariance models were examined using likelihood ratio tests, with statistically 
significant chi-square values indicating that metric or scalar invariance did not hold across 
groups. Mean differences in fidelity by implementation group were examined in the scalar 
invariance model. 

To determine the extent to which the relations between ABISS scores and fidelity varied 
by stage of implementation (Research Question #3), a multiple-group structural equation model 
was fit using the WLSMV estimator, with the measurement models for the ABISS and fidelity 
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factor specified according to the final invariance models in the prior analyses. Equality of the 
regression paths from the ABISS factor to the fidelity factor was investigated using a Wald test, 
and the magnitude and statistical significance of the paths were explored by implementation 
group. 


In all analyses, model fit was evaluated based on conventional criteria (Mueller & 
Hancock, 2010): Comparative Fit Index (CFI) > .95 and Root Mean Square Error of 
Approximation (RMSEA) and its 90% confidence interval < .05. Standard errors were adjusted 
in all models for nesting of schools in districts using the COMPLEX option in M plus, and 
missing data were handled under the assumption that data were missing at random with the 
WLSMV or MLR estimator. Also, for tests of differences in ABISS and fidelity by 
implementation group, standardized mean differences (cl) were reported as an effect size measure 
using the following formula (Hancock, 2001): 


d 






n l + n 2 


( 1 ) 


where Y i and Y 2 are latent means in two groups, n, and n 2 are sample sizes in the groups, and 


Sj 2 and s 2 are variances of the latent factors. 


Results 

Results related to Research Question #1 (What is the ABISS factor structure and is it 
invariant across stage of implementation?) are presented first, followed by results related to 
Research Question #2 (Are there mean differences on the ABISS across schools at different 
stages of implementation?), and then results related to Research Question #3 (Do the relations 
between ABISS scores and fidelity differ by stage of implementation?) are presented. Means and 
standard deviations for the ABISS items by implementation group are presented in Table 2. 
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Factor Structure of the ABISS 

The magnitude of the first compared to the second eigenvalues in EFAs in each 
implementation group [initial implementation: 2.57 to 1.15; full operation: 2.71 to 1.08; 
sustainability: 2.84 to 0.97] supported unidimensionality of the ABISS. The internal consistency 
of the ABISS, based on calculation of ordinal alpha (Zumbo et al., 2007), was .75 in the initial 
implementation group, .76 in the full operation group, and .81 in the sustainability group. 

Comparison of the configural and scalar invariance models indicated that scalar 
invariance did not hold, % (26) = 52.03, p = .001. Inspection of model modification indices 
suggested that item 2 (Competing Initiatives) on the ABISS was most problematic, and 
subsequent model tests indicated that a partially-invariant model with factor loadings and 
thresholds for all items other than item 2 constrained to equality across groups fit as well as the 
configural model, % (20) = 27.09, p = .133. Inspection of the factor loadings for item 2 indicated 
smaller factor loadings for the initial implementation and full operation groups relative to the 
sustainability group. Fit for the partially-invariant model was acceptable on the CFI and near the 
criterion for acceptability on the RMSEA, (35) = 74.48, p < .001, CFI = .965, RMSEA = .068, 
90% Cl [.047, .090]. These results indicated that the partially-invariant model for the ABISS was 
most appropriate to use in subsequent tests of substantive research questions. 

Differences in Perceived Barriers across Implementation Stages 

To address Research Question #2, mean differences on the ABISS were tested in the final 
partially-invariant model. Based on inspection of the ABISS factor means, the sustainability 
group (M = -.20, p = .043, d = .39) had fewer perceived barriers than the initial implementation 
group. There was no statistically significant difference in perceptions of barriers between the full 
operation ( M = -.12 , p = .130, d = .23) and initial implementation groups. 
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Relations of ABISS with Fidelity 

As a pre-condition to assessing the relations of the ABISS with implementation fidelity 
across implementation groups, measurement invariance of the fidelity factor was first tested. 

Measurement invariance of the fidelity factor. Comparisons with the configural 
invariance model indicated that metric, % (6) = 7.67, p = .263, and scalar, y" (12) = 10.72, p = 
.553, invariance held for the fidelity factor across implementation groups. The scalar invariance 
model fit well on all criteria other than the 90% Cl for the RMSEA, yj (18) = 23.28, p = .180, 
CFI = .977, RMSEA = .035, 90% Cl [.000, .071]. Internal consistency of the fidelity factor was 
good, with a composite reliability (Raykov, 1997) of .84. Fidelity was statistically significantly 
higher in the full operation group ( M = .14, p < .001, d = .97) and sustainability group (M = .20, 
p < .001, d = 1.38) compared to the initial implementation group. 

Model with only school demographic controls. Prior to estimating the full model 
assessing the relations between the ABISS and fidelity, a baseline multiple-group model was fit 
with only the school demographic control variables (i.e., middle vs. elementary school, high vs. 
elementary school, enrollment, percentage minority students, percentage of students eligible for 
free or reduced lunch, school location) as predictors of fidelity, using the scalar invariance model 
previously described. Model fit was good on all indicators, y (72) = 86.99, p = .110, CFI = .954, 
RMSEA = .030, 90% Cl [.000, .050]. The school demographic variables explained 10.7% of the 
variance in fidelity in the initial implementation group, with high vs. elementary school (P = - 
.263, p = .001) as the only statistically significant predictor. In the full operation group, the 
demographic variables explained 22.0% of the variance in fidelity, with high vs. elementary 
school (P = -.260, p = .001) and larger enrollments (P = -.260, p = .001) both predicting lower 
levels of fidelity. In the sustainability group, the demographic variables explained 22.6% of the 



BARRIERS TO SUSTAINABILITY OF PREVENTION PRACTICES 


19 


variance in fidelity, with middle vs. elementary (P = -.177, p = .028) and locations in a more 
urban area (P = -.417 , p < .001) both predicting lower levels of fidelity. 

Model with ABISS and school demographic controls. To address Research Question 
#3, a final model was fit to determine the extent to which the ABISS was related to 
implementation fidelity after controlling for school demographic variables. The full model with 
school demographic control variables and the partially-invariant ABISS factor as predictors of 
latent fidelity also fit the data well on all indicators other than chi square, yj (236) = 280.22, p = 
.026, CFI = .957, RMSEA = .028, 90% Cl [.Oil, .040]. As presented in Table 3, the following 
school demographic variables predicted lower levels of fidelity by implementation group: initial 
implementation, high vs. elementary (P = -.237, p = .045); full operation, high vs. elementary (P 
= -.237, p < .001), enrollment (P = -.202, p = .003), percentage minority students (P = -.204,/> = 
.015); and sustainability, middle vs. elementary (P = -.193, p = .013), school location (P = -.428, 
p = .002). In addition, the following school demographic variables predicted greater perceptions 
of barriers on the ABISS by implementation group: initial implementation, percentage of 
students eligible for free/reduced lunches (P = .291, p = .039); full operation, percentage of 
students eligible for free/reduced lunches (P = .277, p = .011); and sustainability, middle vs. 
elementary (P = .261, p = .013), high vs. elementary (P = .334, p = .008). The school 
demographic variables explained 20.8% of the variance in ABISS scores in the initial 
implementation group, 19.9% of the variance in ABISS scores in the full operation group, and 
10.8% of the variance in ABISS scores in the sustainability group. 

A Wald test indicated that the regression paths from the ABISS to fidelity differed by 
implementation group, % (2) = 6.22, p = .045. In the initial implementation (P = -.279, p = .007) 
and full operation (P = -.426, p = .007) groups, greater perceptions of barriers to sustainability on 
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the ABISS were independently associated with lower levels of fidelity. In contrast, perceptions 
of barriers on the ABISS were unrelated to fidelity in the sustainability group (P = -.034, p = 
.769). The ABISS explained an additional 9.0% of the variance in fidelity beyond the school 
demographic variables in the initial implementation group, an additional 20.8% of variance in 
fidelity in the full operation group, and only 2.8% additional variance in fidelity in the 
sustainability group. 

Discussion 

This study evaluated the factor structure of a brief measure of barriers to sustained 
implementation and assessed the extent to which scores on the measure relate to SWPBIS 
fidelity of implementation data across each stage of SWPBIS implementation. With respect to 
the proposed research questions, we first analyzed the factor structure of the ABISS and the 
extent to which measurement invariance held across phases of SWPBIS implementation. Results 
indicated that partial measurement invariance held for the unidimensional ABISS across phases 
of implementation and that the ABISS exhibited adequate internal consistency. Second, we 
examined whether there were mean differences in perceptions of barriers, as measured by the 
ABISS, across schools at different SWPBIS stages of implementation. Results indicated that the 
sustainability group had fewer perceived barriers than the initial implementation group; no 
statistically significant difference was observed in perceptions of barriers between the full 
operation and initial implementation group. Last, we examined the extent to which perceptions 
of barriers relate to SWPBIS fidelity of implementation, controlling for school demographic 
variables. Results indicated that the relations between perceptions of barriers (ABISS) and 
fidelity differed across SWPBIS implementation stages. For the initial implementation and full 
operation groups, greater perceptions of barriers were associated with lower fidelity of 
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implementation; in the sustainability group, perceptions of barriers were unrelated to fidelity. 

The results support the utility of the ABISS as a preliminary measure to identify overall 
levels of perceived barriers to SWPBIS sustained implementation. Although further 
development is necessary before the widespread use of the ABISS as a diagnostic tool, the 
current results are a good first step towards empirically linking barriers and implementation 
fidelity to sustained implementation. The vast majority of studies on barriers to sustainability are 
retrospective accounts of failures to sustain; to our knowledge, the present study is among the 
first empirical studies to examine barriers in relation to objective measures of fidelity of 
implementation. This and future studies can begin to form an empirical basis for decision¬ 
making regarding the identification and remediation of factors hindering implementation and 
sustainability for evidence-based practices. 

The present results highlight the importance of assessing barriers during the beginning 
stages of implementation, but also through full implementation. It is not surprising that reported 
barriers were lowest in the sustainability group, and also unrelated to fidelity of implementation 
in that stage. It is logical to assume that in order to have maintained the practice and reached 
sustainability, schools would have needed to overcome barriers. Successful schools may address 
many potential barriers prior to full implementation. To promote durable change from the outset, 
it is desirable to implement in “fertile ground,” that is, in school environments with fewer 
barriers to implementation and sustainability, or ensure that barriers are addressed prior to (or as 
a condition of) implementation (Rogers, 2003). However, the present results also indicate that 
several less malleable school demographic variables, such as grade level and enrolment, 
accounted for a sizable proportion of fidelity variance. This finding suggests that despite efforts 
to implement in fertile ground, schools may face barriers that are difficult to prevent or 
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eliminate. Understanding how these less malleable factors affect fidelity can assist schools in 
proactively offsetting or minimizing their effects; for example, schools with larger enrollments 
may form smaller committees within the larger SWPBIS staff team to distribute responsibility 
and increase accountability. 

In the Fixsen et al. (2005) stages of implementation model, the program installation phase 
addresses pre-implementation activities to ensure readiness, which can minimize potential 
barriers. Activities such as securing staff time and resources prior to implementation are 
essential for building the infrastructure of a successful practice (Bertram, Blase, & Fixsen, in 
press). However, although barriers may be most pressing to assess and address pre¬ 
implementation, on-going monitoring remains essential. Changes to resources, context, or 
priority can occur at any point in implementation. According to Fixsen et al., maintaining high 
fidelity while responding to these changes characterizes the sustainability stage. 

Although the present results indicate that perceived barriers are related to implementation 
fidelity, it is necessary to understand the importance of barriers relative to other contextual 
features of implementation, such as facilitators. For example, McIntosh et al. (2014) found that 
school team members rated barriers as relatively less important compared to facilitators for both 
initial implementation and sustainability. Barriers and facilitators are inextricably linked, with 
some features often described as representing both a barrier and facilitator. For example, the 
lack of administrator support, staff buy-in, adequate training, and resources are commonly 
reported barriers, with the presence of these features reported as facilitators (Andreou et al., 

2015; Coffey & Homer, 2012; Kincaid et al., 2007; Pinkelman et al., 2015). Even so, previous 
studies have identified uniquely reported facilitators to sustainability; for example, although 
features such as administrator support and staff buy-in were reported as both facilitators and 



BARRIERS TO SUSTAINABILITY OF PREVENTION PRACTICES 


23 


barriers, Kincaid et al. (2007) noted ongoing external support and the integration of the practice 
into the school among the uniquely reported facilitators, and misperceptions or poor knowledge 
about the practice as unique barriers. Future research should further differentiate facilitators 
from barriers and also determine their relative importance for sustained implementation. 

Limitations 

A number of key limitations of the current study are worth noting. First, these findings 
are most applicable to schools implementing SWPBIS. Previous research has shown that 
findings regarding variables related to implementation may be practice-specific, and particular 
aspects of practices may make them more or less robust to barriers (Fixsen et al., 2005). For 
example, schools implementing SWPBIS must often demonstrate readiness before initial training 
begins, and therefore some barriers (e.g., philosophical opposition) may have been addressed 
before a decision to adopt SWPBIS. Second, schools were only included in the study if they had 
completed SWPBIS fidelity assessments and volunteered to complete the ABISS, and thus 
findings may not generalize to schools not conducting external or internal SWPBIS assessments 
of fidelity. Finally, because of the relatively few studies examining barriers empirically, the 
results need replication, especially for approaches other than SWPBIS. 

Potential Implications for Practice and Future Research 

Results from this study suggest that the ABISS adequately serves as a preliminary brief 
measure for assessing barriers to implementation of SWPBIS, though further development and 
expansion are needed prior to its widespread use. Given that overall scale scores were related to 
fidelity of implementation, the specific barriers included may be important for schools to 
consider during initial adoption as well as ongoing implementation of SWPBIS. Barriers such as 
the philosophical opposition of staff members to SWPBIS, competing initiatives already 
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occurring within the school, and staff turnover frequently present great challenges to the 
sustainability of practices within schools. However, utilizing a brief measure to identify the 
existence of such barriers may provide school teams the opportunity to be proactive in their 
response to the existence of barriers by developing plans for how to best address them. For 
example, if a school team finds that they have too many competing initiatives taking place in 
their school, they may work to identify which practices lack evidence, which are not feasible to 
maintain, and which may be braided with other initiatives that address the same common goal. 
Therefore, knowledge of the existence of this barrier may provide the team with a just reason for 
eliminating programs that are ineffective or require a great deal of resources. For barriers that 
are not directly malleable, such as staff turnover, team members may develop a training plan for 
new staff members on core features of their school-wide system that may be delivered at 
different times of the school year. 

The current findings do not identify which specific barriers most strongly predict initial 
or sustained implementation of evidence-based practices. More studies that analyze the 
individual relations of specific barriers to fidelity of implementation across a broader range of 
barriers than are included in the ABISS are needed. For example, the ABISS is strongly 
weighted towards assessing personnel turnover; although this is a common and important barrier, 
the scale is limited in its content of other barriers identified in research such as lack of 
administrator support, staff buy-in, and resources. Future development of the ABISS will focus 
on expansion to more adequately sample the content area in order to serve as a more 
comprehensive assessment of potential barriers. 

The present study provided empirical evidence to support the hypothesis that the overall 
presence of barriers is related to fidelity, but future research may identify methods of 
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determining specific barriers that impact implementation and their relative importance for 
sustainability. Doing so will assist schools in identifying specific areas for prevention and 
remediation of barriers. Finally, our finding that the differential relation of barriers to fidelity by 
the stage of implementation in which schools were at suggests that future research may consider 
the possibility that coaching strategies may need to vary by implementation stage. The supports 
provided by coaches to help schools minimize the impact of barriers on implementation may be 
quite different for schools in the initial stages of adoption of a practice versus school that have 
been implementing for several years. We encourage future research on the examination of how 
effective coaching strategies may be differentiated for schools at these different stages. 
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Table 1 


School Demographics by Implementation Group 


Variable 

Initial 

Implementation Group 

Full Operation 

Sustainability 

% Elementary Schools 

62 

75 

69 

% Middle Schools 

26 

16 

23 

% High Schools 

12 

9 

8 

Mean Enrollment ( SD) 

532 (370) 

532 (305) 

573 (279) 

Mean % Minority Students (SD) 

34 (33) 

39 (31) 

41(25) 

Mean % Students Eligible for 

52 (24) 

48 (23) 

48 (22) 

Free/Reduced Lunches (SD) 




% Schools in Rural Area 

38 

22 

21 

% Schools in Town 

18 

13 

12 

% Schools in Suburb 

17 

36 

45 

% Schools in City 

27 

30 

23 


Sample Size (n) 


180 


329 


195 
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Table 2 

Means and Standard Deviations ofABISS Items by Implementation Group 




Initial 

Full Operation 

Sustainability 


Implementation 





Item 

M 

SD 

M 

SD 

M 

SD 

1. School personnel are 
opposed to SWPBIS 
because it goes against then- 
personal values (e.g., 
“rewarding” students, 
teaching “compliance”). 

1.49 

.62 

1.47 

.68 

1.43 

.68 

2. Other school/district 

2.02 

.91 

1.94 

1.00 

1.96 

1.06 

initiatives (e.g., academic, 
behavior, etc.) are present 
that compete (for time, 
resources or content) with 
SWPBIS. 

3. There are high levels of 
turnover of school 

1.40 

.71 

1.36 

.75 

1.41 

.84 

administrators (i.e., yearly). 

4. There are high levels of 
turnover of school 

1.39 

.70 

1.51 

.85 

1.46 

.80 

personnel who served as 
key leaders (“champions”) 
of SWPBIS (i.e., within 
three years). 

5. There are high levels of 
general school personnel 
turnover (i.e., 50% of staff). 

1.31 

.69 

1.34 

.76 

1.34 

.76 

Sample Size (n) 

180 


329 


195 



Note. Respondents to the ABISS are asked to indicate “To what extent is this statement true for 
your school right now?” 
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Table 3 


Latent Fidelity and ABISS Scores Regressed on School Demographic Variables by 
Implementation Group 


Criterion 

Predictor 

Initial 

P 

Implementation Group 

Full Operation Sustainability 

P P 

Fidelity 

ABISS 

-.279" 

STS 

-.426 

-.034 


Middle vs. Elementary 

-.060 

.004 

* 

-.193 


High vs. Elementary 

* 

-.237 

-.237 

-.137 


Enrollment 

.040 

-.202 

-.156 


Percentage Minority Students 

-.285 

:k 

-.204 

.069 


Percentage Free/Reduced Lunch 

.249 

.072 

-.003 


School Location 

-.041 

-.013 

-.428 

ABISS 

Middle vs. Elementary 

.075 

.156 

>fc 

.261 


High vs. Elementary 

.100 

.127 

.334 


Enrollment 

.108 

.007 

-.129 


Percentage Minority Students 

.005 

.116 

.158 


Percentage Free/Reduced Lunch 

5fc 

.291 

* 

.277 

.034 


School Location 

.217 

.147 

.026 

Sample Size (n) 

180 

329 

195 


sE sEsE iE"iE"iE 

Note. p<. 05, p < .01, p < .001 



